Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for type checking with mypy plugins #121

Closed
wants to merge 5 commits into from

Conversation

flaeppe
Copy link

@flaeppe flaeppe commented Apr 19, 2024

I'm not totally sure if this approach always works or if there's stuff that I'm missing, but it seems to work. I'll try to explain myself here.

The change utilises pip's --target(docs) argument to point out where to install a project's additional dependencies, passed in via pip_cmd. This replaces the virtual env creation for each project, since that shouldn't be necessary (I think?).

In addition to what's mentioned above we start setting the PYTHONPATH env variable(docs), to point out an additional search path for module files, namely the site-packages directory with the additional dependencies inside the project's virtualenv.

That should allow discovery of project dependencies while keeping the installed mypy versions independent and reusable.

Outdated

All this isn't included, but I was thinking that it should be possible to use pip install --target for both mypy installations as well, instead of a virtual env. That leads to having 1 python/pip executable, 1 directory for new_mypy, 1 directory for old_mypy and then all project dependency installations. Although I'm unsure what --python-executable= does and if it could be affected. I'm also not sure how to handle compiled mypy.

Essentially what'll happen with the above changes is that we'll have a directory structure like e.g.

repo-dir/
├── new_mypy
│   └── bin
│       └── mypy
├── old_mypy
│   └── bin
│       └── mypy
├── projects
│   ├── A
│   └── B
└── python
    └── bin
        └── python

We run mypy with something corresponding to

PYTHONPATH="repo-dir/new_mypy:repo-dir/projects/A" repo-dir/new_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ...
PYTHONPATH="repo-dir/old_mypy:repo-dir/projects/A" repo-dir/old_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ...

Here's some example debug output:

...
/tmp/mypy_primer/projects/_django-choicefield_venv/bin/pip install . django-stubs pytest         in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/new_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/new_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/old_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/old_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
/tmp/mypy_primer/new_mypy/venv/bin/mypy on django-choicefield took 4.96s
/tmp/mypy_primer/old_mypy/venv/bin/mypy on django-choicefield took 4.98s
...

mypy_primer/model.py Outdated Show resolved Hide resolved
mypy_primer/model.py Outdated Show resolved Hide resolved
mypy_primer/model.py Outdated Show resolved Hide resolved
Copy link
Owner

@hauntsaninja hauntsaninja left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

I don't quite understand what the new global env is for. Other than that, this PR looks great and does the right thing: we should continue to setup a) a venv for mypy itself, b) project specific venvs. And then use PYTHONPATH (or other) shenanigans so that mypy can import from the project specific venv. We should also add an assert that some version of mypy itself isn't in the project specific venv, otherwise things will get confusing.

@flaeppe
Copy link
Author

flaeppe commented Apr 29, 2024

Thanks for the review, I'm gonna try to move back to using project specific venvs, see if I can get it to work.

The global env was added to have a python/pip binary to reach to when I removed the project specific venvs.

I'll see if I can get a reasonable assert in there too, perhaps just something naive like looking for bin/mypy in a project specific venv is enough.

@flaeppe
Copy link
Author

flaeppe commented Apr 29, 2024

I've updated to align with your suggestions. Let me know if there's anything else I've missed or should update.

Though I can't say I follow why each project needs its own environment. I was thinking it would be preferable to have a virtualenv for each mypy installation and then do a "distributed" install for each project (pip install --target) and via PYTHONPATH= "mount" in the relevant dependencies for each project run. But I might definitely miss something.

@hauntsaninja
Copy link
Owner

Thanks again for the PR, sorry that I lost track of it!
This developed a few merge conflicts, I fixed in 83721d5. I also altered the approach slightly, I use a .pth file to append to sys.path, so that project venv will never cause conflicts with mypy and its dependencies.

@flaeppe flaeppe deleted the feature/plugins branch July 13, 2024 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

do a plugin hack How to add dependencies with plugins?
2 participants